Real-time process and technology using image processing to maintain and ensure viewer comfort during capture, live transmission, and post-production of stereoscopic 3d imagery

ABSTRACT

This process and invention may be used for “live” (on-line, recorded, real-time, or transmitted) stereoscopic image generation, post-production, and viewing to ensure the imagery, from the cameras through the systems to the final screens, maintains an acceptable level of viewer comfort and actively causes an action if an image becomes “uncomfortable” to view. The invention will use image-processing hardware and software to monitor stereoscopic 3D image data, and will generate an output response based on qualifying the imagery. The output responses include generating an alarm to alert technical crew members to a “discomfort” condition so they may take corrective action. The alarm may be an audio or visual alert, and warn of the type of discomfort detected. The alarm condition may trigger the automatic control of a video switching device, which would immediately route an appropriate “comfortable” input source to the output. The alarm condition may duplex a single camera or playback signal into an identical stereo pair, which will be 2D, but will be comfortable to view.

CROSS-REFERENCE TO RELATED APPLICATIONS

Any and all priority claims identified in the Application Data Sheet, orany correction thereto, are hereby incorporated by reference under 37CFR 1.57.

FIELD OF THE INVENTION

The present invention relates generally to stereoscopic 3D camera andviewing systems.

BACKGROUND OF THE INVENTION

For many years stereoscopic 3D movies have been made. Some have beengood, but many others have been bad. The badly made 3D movies have beenmainly the symptom of inadequate equipment, or the lack of knowledge ofgood stereography. The effectively creates discomfort by the viewer, aswell as having a negative impact on the 3D industry.

In this digital age, it is now feasible to create technologicalsolutions to this dilemma.

This invention describes a process to ensure a comfortable Stereoscopic3D viewing experience, whereby the viewer will observe natural 3Dimagery without the psychovisual effects of eye strain, fatigue, ordiscomfort.

SUMMARY OF THE INVENTION

The process of this invention is performed by extracting usefulinformation from visual content, by means of image-processing the 3Dimagery from a stereoscopic 3D camera system, and provides an outputresponse based on logic decisions and pre-defined thresholds.

The output response includes generating alarms to alert technical crewmembers of a “discomfort” condition so they may take corrective action.These alarms may be an audio or visual alert, and warn of the type ofdiscomfort detected. The alarm condition may also trigger the automaticcontrol of a video switching device, which would immediately route anappropriate “comfortable” input source to the output. The matrixswitcher requires a common banking capability, because stereo pairs arerouted simultaneously to the output.

The alarm condition may duplex a single camera or playback signal intoan identical stereo pair, which will be 2D, but comfortable to view.

(The “comfortable” input source may be a pre-defined or known“comfortable” 3D camera imagery, or other video content from a playbackdevice.)

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a typical signal path of a “live” stereoscopic 3Dinfrastructure, including the processing units involved.

FIG. 2 shows a typical Capture-side detail, in the form of a blockdiagram.

DETAILED DESCRIPTION

One embodiment of this invention (FIG. 1) includes a complete “live”infrastructure, consisting of the following signal path: a stereoscopic3D camera rig/s, an image processing unit to ensure comfortable 3Dcapture, encoding equipment, transmission equipment at the capture side,reception equipment at the viewing side/s, decoding equipment, animage-processing unit to ensure the reception conforms to comfortable 3Dviewing, and a 3D display apparatus such as a 3D projector/s.

During capture, the imagery may be uncomfortable to view due to manyvariables and reasons, and these are quantified by image processingelectronics, which generates alarms based on pre-defined criteria, andalarm thresholds.

When using a remote viewing location, where the video content istransmitted “live” from the 3D camera location, additional imageprocessing electronics is required at the viewing location to ensure theprojectors are fed “comfortable” 3D imagery, in case there is failure ofthe captured quality, during the transmission. The imagery can becorrupted, missing, noisy, or out of sync. This electronics system onthe receiving side would use a subset of the image-processing describedabove, and use a matrix switched output, to guarantee the projectors arefed “comfortable” 3D content.

This process may also be used in post-production (off-line), to ensurerecorded media maintains comfortable 3D imagery, and generates similaroutput responses appropriate for an edit session, and is especiallyuseful for finding transition points that are “comfortable” forclose-matching 3D.

This invention uses a mathematical model to quantify acceptable rangesof comfort.

The capture-side (FIG. 2) input into to the system may include:

1) Imagery from the left camera of the stereoscopic 3D camera rig, to beimage processed.

2) Imagery from the right camera of the stereoscopic 3D camera rig, tobe image processed.

3) Focus metadata from the 3D rig.

4) Iris metadata from the 3D rig.

5) Zoom metadata from the 3D rig.

6) Inter-Ocular metadata from the 3D rig.

7) Convergence metadata from the 3D rig.

8) Screen dimensions. (for TV, theater, IMAX, etc)

9) Distance range between this screen and the viewers.

10) Acceptable image horizontal disparity, which may be expressed as apercentage of total image size.

11) Fusional range.

12) Alarm thresholds:

a) Gross difference (non-“fusable”)

b) Focus disparity

c) Luminance disparity

d) Chrominance disparity

e) Magnification disparity

f) Telecentricity disparity

g) “Broken frame” acceptance level

h) Vertical content weighting factor

i) Vertical disparity (expressed as number of lines, angle, orPercentage of screen height)

To generate an alarm, the system's image-processing function will firstlook for obvious image errors such as missing video from a camera orcameras, or out-of-sync video, either sub-frame or multiple frame.

Then obvious lens mismatch is processed. Focus disparity is calculated,where the image-processing algorithm includes edge detection and/or ahigh-pass filtering to narrow in on the highest frequency detail of thechart. Luminance disparity (created by iris, gamma, black-level, knee,etc.) is calculated, where the image-processing algorithm includes imagesubtraction and/or image correlation. Chrominance disparity (hue,saturation) is calculated, where the image-processing algorithm includescolor matrix conversion, and image correlation. Alarms are generated ifthe mismatches exceed pre-defined thresholds.

Then by using disparity mapping, by block and pixel methods, a depth mapis created. This is done using a standard neural-net process, wherestrong links are magnified (for adjacency) by parallel vector analysisto find stereoscopically “fusable” subject matter. Broken links in theneural-net will be determined to be caused by either “breaking frame” onthe boundary of the images, or from stereoscopic occlusion within theimages. The “breaking-frame” condition has an adjustable acceptancelevel, or alarm threshold.

“Blob analysis” algorithms are used to combine any linked “fusable”subject matter into bigger “blobs”.

The amount of “fusable” subject matter, as an area ratio of the fullscreen size, is used to determine if there is a gross difference fromboth camera views, which may be caused by something obstructing the viewof one camera. If this gross difference is sufficient to exceed thealarm threshold, an alarm condition will be generated.

The “blobs are analyzed for and magnification disparity (zoom mismatch),and telecentricity mismatch, upon which an alarm will be generated ifthese mismatches exceed the alarm thresholds.

The range of all angles to the boundaries of “fusable” subject matter,or “fusional range” are calculated, and if any angle exceeds the alarmthreshold, an alarm will be generated. These angles to the “fusable”subject matter are performed in the horizontal plain, as this is thenatural stereoscopic horizontal disparity. Excessive horizontaldisparity, either towards viewer divergence or excessive convergence,will generate an alarm.

The search range of the neural net will include several lines above andbelow the present search range, to extract possible vertical orrotational disparity, upon which an alarm will be generated if thevertical disparity is found to exceed the alarm threshold, and takesinto account the screen size.

The background will be searched for a concentration of vertical content(such as lamp posts, or a fence line). A Fourier transform is performedin the horizontal direction to extract this in the frequency domain.This area of the image will be considered less stereoscopically“fusable”, and weighted accordingly, taking into account other “fusable”subject matter. An alarm will be generated if it exceeds a pre-definedthreshold.

Finally, the remaining uncategorized areas will be deemed occlusionareas, and will be ignored, because they are naturally stereoscopic.

The alarm condition may also trigger the automatic control of a videoswitching device, which would immediately route an appropriate“comfortable” input source to the output.

1. (canceled)
 2. A system, comprising: a stereoscopic camera rigcomprising a stereoscopic camera; an image capture processor operablyconnected to the stereoscopic camera, wherein the image captureprocessor executes instructions to implement real time image processingof stereoscopic image inputs and further executes decision logicinstructions to determine an alarm condition based on the real timeimage processing; one or more than one alarm operably connected to theimage capture processor and a display image processor; one or more thanone video switch operably connected to the display image processor; anda display connected to the one or more than one video switch, whereinthe stereoscopic image inputs comprise one or more of: one or moreimages from a left camera of the stereoscopic camera; one or more imagesfrom a right camera of the stereoscopic camera; focus metadataassociated with the rig; iris metadata associated with the rig; zoommetadata associated with the rig; inter-ocular metadata associated withthe rig; convergence metadata associated with the rig; one or morescreen dimensions associated with the display; one or both of a distancerange between a screen of the display and one or more viewers and adistance between the screen of the display and one or more viewers;image horizontal disparity, that can be expressed as a percentage oftotal image size; fusional range; and one or more alarm thresholds. 3.The system of claim 2, wherein the system further comprises one or moreof: an encoder operably connected to the image capture processor; atransmitter operably connected to the encoder; a receiver operablyconnected to the transmitter; and a decoder operably connected to thereceiver; wherein the display image processor is operably connected tothe decoder;
 4. The system of claim 2, wherein the stereoscopic cameracomprises a and lens motor control connected to the rig and a userinterface connected to the rig.
 5. The system of claim 2, wherein thescreen dimensions comprise an average of an expected screen size.
 6. Thesystem of claim 2, wherein the screen dimensions comprise an expectedlargest screen size.
 7. The system of claim 2, wherein the alarmthresholds are selected from the group consisting of gross(non-“fusable”) difference, focus disparity, luminance disparity,chrominance disparity, magnification disparity, telecentricitydisparity, “broken frame” acceptance levels, vertical content weightingfactors and vertical disparity values expressed as number of lines,angle, or percentage of screen height.
 8. The system of claim 2, whereinthe one or more than one alarm outputs a visual alarm.
 9. The system ofclaim 2, wherein the one or more than one alarm outputs an audiblealarm.
 10. The system of claim 2, wherein the one or more than one videoswitch causes a switch between displaying normal or alternative video.11. The system of claim 2, wherein the one or more than one video switchcomprises a matrix switcher having common banking capability so thatstereo pairs are routed simultaneously to the output.